The Ambulant Annotator: Medical Multimedia Annotations on Tablet PC’s

A new generation of tablet computers has stimulated end-user interest on annotating documents by making pen-based commentary and spoken audio labels to otherwise static documents. The typical application scenario for most annotation systems is to convert existing content to a (virtual) image, capture annotation mark-up, and to then save the annotations is a database. While this is often acceptable for text documents, most multimedia documents are timesensitive and can be dynamic: content can change often depending on the types of audio/video data used. Our work looks at expanding the possibilities of annotation by integrating annotations onto timed basis media. This paper discusses the AMBULANT Annotation Architecture. We describe requirements for multimedia annotations, the multimedia annotation architecture being developed at CWI, and initial experience from providing various classes of temporal and spatial annotation within the domain of medical documents.. Introduction This article presents an approach to attaching multimedia commentary to multimedia documents. Such commentary is often called an annotation. Annotations have been the subject of much research in the past decade, mostly in the context of adding written or ink commentary to text documents [7],[10],[13],[15]. Unfortunately, much of the work on text annotation has two fundamental limitations: the underlying document model cannot accommodate media that is temporal rather than static and the annotation facilities have only a limited content-based relationship to the underlying document. The focus of our work is extending document annotation facilities so that comments can be associated with temporal documents (such as those containing audio and video media) in such a way that the annotations can be defined in the logical context of the underlying story. In the sections below, we start with a discussion of the requirements of a multimedia annotation architecture. We next describe the how our SMIL-based annotation approach can be applied to a class of media documents (in our case, medical documents containing text, audio, image and video media objects) and how we are implementing this functionality in the CWI Ambulant Annotator. We conclude with some directions for our future work. Requirements for a Multimedia Annotation System There have been very many approaches used to support the annotation of electronic documents. Most of these approaches have been targeted to special user groups (such as students or editors) and nearly all have been based on the mark-up of static media such as text and images. The annotations have had several goals: to provide meta-data that can be used for information classification [16], to provide searching/indexing facilities to support information retrieval [9] and to support informal and formal semantic mark-up of document content [12]. Nearly all of these systems define annotations as a collections of keywords, even though pen-based markup has historically been a more useful ways of attaching commentary to documents. A major step forward in the process of making annotation a first-class user-interface feature came with the broad introduction of Microsoft’s Tablet PC. The Tablet PC is a full-featured portable computer that can be used to execute any Windows or Linux application; it’s primary added-value feature is a writable display and an electronic pen. Early versions of the Tablet PC have approximately 1-Ghz processors and a full-color display with a resolution of 1028x768 pixels. Unlike PDAs like the Palm or the PocketPC, the Tablet PC is a ‘real’ computer, with a ‘real’ display that can be used for running a wealth of ‘real’ applications. Most Tablet PCs have wireless and fixed networking capabilities and 30 gigabyte disks. Some have detachable keyboards, others use a slate-model. Limitations of the Windows Journal Annotation System Early tablet computers come with a pen-annotation system developed by Microsoft under the name Windows Journal. The Journal’s interaction model is typical for may annotation systems that are built on a ‘virtual image’ model. This model takes a base document and treats it as a background layer on top of which annotations are made. In complex systems, it is often possible to have multiple layers of annotations, some of which can be independently rendered, but all of these are applied to the spatial mapping of a single instance of the source document. The limitations of the Windows Journal stem, in part, from its strict adherence to a pen-and-paper model: you markup one fixed instance of a document rather than the document itself. This is a fairly common restriction in annotation systems, not so much because of the influence of years of suffering under irate librarians, but because the annotation systems are built on-top of media rendering systems and not within media rendering systems. Our work takes a fundamentally different approach. Required Multimedia Extensions The goal of our research is to produce an annotation environment for multimedia documents. Rather than using a virtual image model, our work seeks to capitalize on the underlying multimedia structure of a document by allowing annotators to add annotations to continuous and discrete media in the context of the original presentation. The design features for our environment are listed in the following table.